Cloud storage service provides a stable, secure, efficient and strongly scalable storage service. It not only directly provides services to users, but also provide a strong support to upper-layer services; therefore, it is a very important basic service in public cloud services.
An abundance number of cloud storage products are available in the market. With constant change of the market or the demands of cloud storage users, the cloud storage user may select different cloud storage services to use. This generates a demand of migrating data between different cloud storages, e.g., migrating data from Alibaba Cloud OSS to Baidu cloud BOS. For data migration between cloud storage systems, currently, different tools are all developed for specific systems and specific scenarios, e.g., the migration tool qrsync of Qiniu Cloud, the importer of Alibaba Cloud OSS. These migration tools read data from a specified source end, write them to a destination end, and then verify whether the data are correctly written into the destination end, thereby completing an action of data migration. Besides, based on the approaches of the various migration tools above, the data need to be written into a disk once, thereby generating an unnecessary disk I/O, and it is highly difficult to control the migration speed and scale.
However, in the cloud storage system, the data amount is tremendous, and a lot of uncertainties arise because the data migration between cloud storage systems is essentially interactivity over a network. Therefore, some important aspects should be considered during the process of data migration:
1) when the data amount is considerable, how to complete data migration at a fastest speed without affecting the current system, i.e., demands of high performance, controllable speed, and strong scalability;
2) capable of automatically/accurately implementing data migration under various complex uncertain circumstances such as network, i.e., demands of automation and strong verify;
3) a variety of cloud storage products are available on the market. However, if a migration tool supporting the above two aspects is developed for data migration between every two cloud storage systems, the development work will be repetitive and intensive, i.e., demands of universality of the migration framework.
Based on the considerations of the above aspects, the data migration between cloud storage systems faces huge challenges in performance, flexibility, scalability, automation, data verify and frame universality.